System and method for deep labeling

ABSTRACT

An apparatus for contextual execution comprises a processor, and a memory containing instructions, which when executed by the processor, cause the apparatus to receive, from a user terminal, a control input associated with an intent, obtain location data associated with a location of the user terminal, and determine a scored set of execution options associated with the control input. Further, the instructions, when executed by the processor cause the apparatus to obtain a contextual label associated with the location data, the label determined based on the application of one or more adapted pretrained deep learning models to the location data.

CROSS-REFERENCE TO RELATED APPLICATION AND CLAIM OF PRIORITY

This application claims priority under 35 U.S.C. § 119(e) to U.S.Provisional Patent Application No. 62/471,000 filed on Mar. 14, 2017.The above-identified provisional patent application is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

This disclosure relates generally to improving the operation ofcomputing platforms by providing contextual execution of control inputsassociated with user intents. More specifically, this disclosure relatesto deep-learning based crowd-assisted systems for contextual labeling,including, without limitation, systems for deep labeling.

BACKGROUND

Improved sensor technologies and machine-based voice and imagerecognition technologies provide important building blocks for futureprocessor based apparatus which able to operate in response to usercommands which, through their reliance intents and inferences, mimic thecontext-dependent ways in which humans communicate. However, thetechnical challenges associated with developing such contextuallyintelligent apparatus include, without limitation, developing corpusesof contextual labels associated with locations or other sources ofcontext for user commands. For example, manual labelling of geographicdata may be inadequate, given the sheer number of locations to belabelled and the limited accuracy of humans performing labelling.Further, the technical challenges associated with developingcontextually intelligent apparatus include the fact that many machinelearning techniques, such as deep neural networks (DNN), may requiretraining on very large datasets. However, for many applications, asuitable dataset of sufficient size for training a DNN may not beavailable. Embodiments as disclosed and described herein are addressedto meeting these and other technical challenges associated withdeveloping apparatus with contextual intelligence.

SUMMARY

This disclosure provides systems and methods for deep labeling.

In a first embodiment, an apparatus includes a processor and a memorycontaining instructions, which when executed by the processor, cause theapparatus to receive one or more pretrained deep learning models, eachpretrained deep learning model associated with a source domain, receiveimage data to be labeled, and input the received image data to each ofthe one or more pretrained deep learning models. Further, theinstructions, when executed by the processor, cause the processor toperform an adaptation on one or more of the pretrained deep learningmodels, provide, from each of the pretrained deep learning models, anoutput in a target domain, provide an ensemble output, the ensembleoutput comprising labels for the image data determined based on theoutputs from each of the pretrained deep learning models, and when thetarget domain is not completely covered by the source domain associatedwith a pretrained deep learning model of the one or more pretrained deeplearning models, perform transfer learning on the pretrained deeplearning model.

In a second embodiment, a method for contextual labeling of image dataincludes receiving, one or more pretrained deep learning models, eachpretrained deep learning model associated with a source domain,receiving image data to be labeled, and inputting the received imagedata to each of the one or more pretrained deep learning models.Additionally, the method includes performing an adaptation on one ormore of the pretrained deep learning models, providing, from each of thepretrained deep learning models, an output in a target domain for thepretrained deep learning model, providing an ensemble output, theensemble output comprising labels for the image data determined based onthe outputs from each of the pretrained deep learning models, and whenthe target domain is not completely covered by the source domainassociated with a pretrained deep learning model of the one or morepretrained deep learning models, perform transfer learning on thepretrained deep learning model.

In a third embodiment, a non-transitory computer-readable mediumincludes program code, which when executed by a processor, causes anapparatus to receive one or more pretrained deep learning models, eachpretrained deep learning model associated with a source domain, receiveimage data to be labeled, and input the received image data to each ofthe one or more pretrained deep learning models. The program code, whenexecuted by the processor, further causes the apparatus to perform anadaptation on one or more of the pretrained deep learning models,provide, from each of the pretrained deep learning models, an output ina target domain for the pretrained deep learning model, provide anensemble output, the ensemble output comprising labels for the imagedata determined based on the outputs from the pretrained deep learningmodels, and when the target domain is not completely covered by thesource domain associated with a pretrained deep learning model of theone or more pretrained deep learning models, perform transfer learningon the pretrained deep learning model.

In a fourth embodiment, an apparatus for contextual execution comprisesa processor, and a memory containing instructions, which when executedby the processor, cause the apparatus to receive, from a user terminal,a control input associated with an intent, obtain location dataassociated with a location of the user terminal, and determine a scoredset of execution options associated with the control input. Further, theinstructions, when executed by the processor, cause the apparatus toobtain a contextual label associated with the location data, the labeldetermined based on the application of one or more adapted pretraineddeep learning models to the location data, rescore the set of executionoptions associated with the control input based on the contextual label,and provide the highest scored execution option to a processor of theuser terminal.

In a fifth embodiment, a method for contextual execution includesreceiving, from a user terminal, a control input associated with anintent, obtaining location data associated with a location of the userterminal, and determining a scored set of execution options associatedwith the control input. The method further includes obtaining acontextual label associated with the location data, the label determinedbased on the application of one or more adapted pretrained deep learningmodels to the location data, rescoring the set of execution optionsassociated with the control input based on the contextual label andproviding the highest scored execution option to a processor of the userterminal.

In a sixth embodiment, a non-transitory computer-readable mediumincludes program code, which when executed by a processor, causes anapparatus to receive, from a user terminal, a control input associatedwith an intent, obtain location data associated with a location of theuser terminal, and determine a scored set of execution optionsassociated with the control input. The program code, when executed bythe processor, further causes the apparatus to obtain a contextual labelassociated with the location data, the label determined based on theapplication of one or more adapted pretrained deep learning models tothe location data, rescore the set of execution options associated withthe control input based on the contextual label, and provide the highestscored execution option to a processor of the user terminal.

Other technical features may be readily apparent to one skilled in theart from the following figures, descriptions, and claims.

Before undertaking the DETAILED DESCRIPTION below, it may beadvantageous to set forth definitions of certain words and phrases usedthroughout this patent document. The term “couple” and its derivativesrefer to any direct or indirect communication between two or moreelements, whether or not those elements are in physical contact with oneanother. The terms “transmit,” “receive,” and “communicate,” as well asderivatives thereof, encompass both direct and indirect communication.The terms “include” and “comprise,” as well as derivatives thereof, meaninclusion without limitation. The term “or” is inclusive, meaningand/or. The phrase “associated with,” as well as derivatives thereof,means to include, be included within, interconnect with, contain, becontained within, connect to or with, couple to or with, be communicablewith, cooperate with, interleave, juxtapose, be proximate to, be boundto or with, have, have a property of, have a relationship to or with, orthe like. The term “controller” means any device, system or part thereofthat controls at least one operation. Such a controller may beimplemented in hardware or a combination of hardware and software and/orfirmware. The functionality associated with any particular controllermay be centralized or distributed, whether locally or remotely. Thephrase “at least one of,” when used with a list of items, means thatdifferent combinations of one or more of the listed items may be used,and only one item in the list may be needed. For example, “at least oneof: A, B, and C” includes any of the following combinations: A, B, C, Aand B, A and C, B and C, and A and B and C.

Moreover, various functions described below can be implemented orsupported by one or more computer programs, each of which is formed fromcomputer readable program code and embodied in a computer readablemedium. The terms “application” and “program” refer to one or morecomputer programs, software components, sets of instructions,procedures, functions, objects, classes, instances, related data, or aportion thereof adapted for implementation in a suitable computerreadable program code. The phrase “computer readable program code”includes any type of computer code, including source code, object code,and executable code. The phrase “computer readable medium” includes anytype of medium capable of being accessed by a computer, such as readonly memory (ROM), random access memory (RAM), a hard disk drive, acompact disc (CD), a digital video disc (DVD), or any other type ofmemory. A “non-transitory” computer readable medium excludes wired,wireless, optical, or other communication links that transporttransitory electrical or other signals. A non-transitory computerreadable medium includes media where data can be permanently stored andmedia where data can be stored and later overwritten, such as arewritable optical disc or an erasable memory device.

Definitions for other certain words and phrases are provided throughoutthis patent document. Those of ordinary skill in the art shouldunderstand that in many if not most instances, such definitions apply toprior as well as future uses of such defined words and phrases.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of this disclosure and its advantages,reference is now made to the following description, taken in conjunctionwith the accompanying drawings, in which:

FIG. 1 illustrates an example of an apparatus according to thisdisclosure;

FIG. 2 illustrates an example of a network context for implementing adeep-learning based, crowd-assisted system for contextual labelingaccording to this disclosure;

FIG. 3 illustrates an example of a pipeline which includes a modeladaptation schema and ensemble of adapted models according to thisdisclosure;

FIG. 4 illustrates operations of a method for performing adaptation ofpretrained deep learning models to obtain an ensemble output accordingto this disclosure;

FIG. 5 illustrates operations of a method for performing a deep-learningbased, crowd-assisted system for location labeling according to thisdisclosure;

FIG. 6 illustrates aspects of a pipeline of deep learning models forgenerating an ensemble output according to this disclosure;

FIGS. 7A-7C illustrate performance results of a test of an apparatusimplementing the pipeline described in FIG. 6;

FIG. 8 illustrates operations of a method for generating alocation-device matrix according to certain embodiments of thisdisclosure;

FIG. 9 illustrates aspects of building a location device matrix based ona bipartite graph according to certain embodiments of this disclosure;and

FIG. 10 illustrates aspects of a method for rescoring execution optionsaccording to certain embodiments of this disclosure.

DETAILED DESCRIPTION

FIGS. 1 through 10, discussed below, and the various embodiments used todescribe the principles of this disclosure in this patent document areby way of illustration only and should not be construed in any way tolimit the scope of the disclosure. Those skilled in the art willunderstand that the principles of this disclosure may be implemented inany suitably configured processing platform.

FIG. 1 illustrates one example of an apparatus 100 according to certainembodiments of this disclosure. The embodiment of apparatus 100illustrated in FIG. 1 is for illustration only, and other configurationsare possible. However, suitable devices come in a wide variety ofconfigurations, and FIG. 1 does not limit the scope of this disclosureto any particular implementation of an apparatus.

As shown in FIG. 1, the apparatus 100 includes a communication unit 110that may include, for example, a radio frequency (RF) transceiver, aBluetooth® transceiver, or a Wi-Fi® transceiver, etc., transmit (TX)processing circuitry 115, a microphone 120, and receive (RX) processingcircuitry 125. The apparatus 100 also includes a speaker 130, a mainprocessor 140, an input/output (I/O) interface (IF) 145, input/outputdevice(s) 150, and a memory 160. The memory 160 includes an operatingsystem (OS) program 161, one or more applications 162 and one or moremodels 169.

Applications 162 can include games, social media applications,applications for geotagging photographs and other items of digitalcontent, virtual reality (VR) applications, augmented reality (AR)applications, operating systems, device security (e.g., anti-theft anddevice tracking) applications or any other applications which accessresources of apparatus 100, the resources of apparatus 100 including,without limitation, speaker 130, microphone 120, input/output devices150, and additional resources 180. Further, applications 162 may includean intelligent assistant application 163, an image recognitionapplication 165 and a voice recognition application 167. According tovarious embodiments, intelligent assistant application 163 operates asan orchestrator and execution manager for other applications 162operating on apparatus 100. For example, intelligent assistant 163 mayreceive outputs (such as data or method calls) from voice recognitionapplication 167 and forward them as inputs to another application, suchas an internet browser or data service application (for example, aweather application). Thus, intelligent assistant 163 can, for example,orchestrate the apparatus' response to certain voice activated commandsor requests, such as a user saying, “What's the weather?” or “Turn onliving room lights.”

The communication unit 110 may receive an incoming RF signal, forexample, a near field communication signal such as a Bluetooth® orWi-Fi® signal. The communication unit 110 can down-convert the incomingRF signal to generate an intermediate frequency (IF) or baseband signal.The IF or baseband signal is sent to the RX processing circuitry 125,which generates a processed baseband signal by filtering, decoding, ordigitizing the baseband or IF signal. The RX processing circuitry 125transmits the processed baseband signal to the speaker 130 (such as forvoice data) or to the main processor 140 for further processing (such asfor web browsing data, online gameplay data, notification data, or othermessage data).

The TX processing circuitry 115 receives analog or digital voice datafrom the microphone 120 or other outgoing baseband data (such as webdata, e-mail, or interactive video game data) from the main processor140. The TX processing circuitry 115 encodes, multiplexes, or digitizesthe outgoing baseband data to generate a processed baseband or IFsignal. The communication unit 110 receives the outgoing processedbaseband or IF signal from the TX processing circuitry 115 andup-converts the baseband or IF signal to an RF signal for transmission.

The main processor 140 can include one or more processors or otherprocessing devices and execute the OS program 161 stored in the memory160 in order to control the overall operation of the apparatus 100. Forexample, the main processor 140 could control the reception of forwardchannel signals and the transmission of reverse channel signals by thecommunication unit 110, the RX processing circuitry 125, and the TXprocessing circuitry 115 in accordance with well-known principles. Insome embodiments, the main processor 140 includes at least onemicroprocessor or microcontroller.

Additionally, in some embodiments operating system 161 is capable ofproviding “secure world” and “normal world” execution environments forapplications 162.

The main processor 140 is also capable of executing other processes andprograms resident in the memory 160. The main processor 140 can movedata into or out of the memory 160 as required by an executing process.In some embodiments, the main processor 140 is configured to execute theapplications 162 based on the OS program 161 or in response to inputsfrom a user or applications 162. Applications 162 can includeapplications specifically developed for the platform of apparatus 100,or legacy applications developed for earlier platforms. The mainprocessor 140 is also coupled to the I/O interface 145, which providesthe apparatus 100 with the ability to connect to other devices such aslaptop computers and handheld computers. The I/O interface 145 is thecommunication path between these accessories and the main processor 140.

The main processor 140 is also coupled to the input/output device(s)150. The operator of the apparatus 100 can use the input/outputdevice(s) 150 to enter data into the apparatus 100. Input/outputdevice(s) 150 can include keyboards, touch screens, mouse(s), trackballs or other devices capable of acting as a user interface to allow auser to interact with apparatus 100. In some embodiments, input/outputdevice(s) 150 can include a touch panel, a virtual reality headset, a(digital) pen sensor, a key, or an ultrasonic input device.

Input/output device(s) 150 can include one or more screens, which can bea liquid crystal display, light-emitting diode (LED) display, an opticalLED (OLED), an active matrix OLED (AMOLED), or other screens capable ofrendering graphics.

The memory 160 is coupled to the main processor 140. According tocertain embodiments, part of the memory 160 includes a random accessmemory (RAM), and another part of the memory 160 includes a Flash memoryor other read-only memory (ROM). In the non-limiting example of FIG. 1,memory 160 includes one or more models 169. In certain embodiments,models 169 comprise pretrained deep learning models, such as AlexNet.Depending on the storage capacity, processing power, and operationalexpectations (for example, battery life, operating temperature and soforth) of apparatus 100, models 162 may comprise lightweight versions ofdeep learning models.

Although FIG. 1 illustrates one example of an apparatus 100. Variouschanges can be made to FIG. 1. For example, according to certainembodiments, apparatus 100 can further include a separate artificialintelligence processing unit 170 (AI PU) or other processor adaptedheavily multithreaded processing applications.

According to certain embodiments, apparatus 100 includes a variety ofadditional resources 180 which can, if permitted, be accessed byapplications 162. According to certain embodiments, resources 180include an accelerometer or inertial motion unit 182, which can detectmovements of the electronic device along one or more degrees of freedom.Additional resources 180 include, in some embodiments, a user's phonebook 184, one or more cameras 186 of apparatus 100, and a globalpositioning system 188. In the non-limiting example of FIG. 1,additional resources 180 may operate to provide location data about alocation (including, for example, GPS coordinates for the location,temperature information, altitude information, or other informationwhich may be used as crowd-sensed input data for models 169. Forexample, resources 180 includes a calendar listing major holidays, andmodels 169 includes adapted pretrained deep learning models forrecognizing features in image data and determining a contextual label(for example, a label assigning a class (e.g., “supermarket” or “shoestore” to the location where the image was captured). Calendarinformation indicating that an image was captured on Christmas Day(December 25) may be used to adjust the probabilities of labels assignedto an image with an abundance of the color red.

Although FIG. 1 illustrates one example of an apparatus 100 forimplementing monitoring of suspicious application access, variouschanges may be made to FIG. 1. For example, the apparatus 100 couldinclude any number of components in any suitable arrangement. Ingeneral, devices including computing and communication systems come in awide variety of configurations, and FIG. 1 does not limit the scope ofthis disclosure to any particular configuration. While FIG. 1illustrates one operational environment in which various featuresdisclosed in this patent document can be used, these features could beused in any other suitable system.

FIG. 2 illustrates an example of a network context 200 for implementinga deep-learning based, crowd-assisted system for contextual labelingaccording to this disclosure.

According to various embodiments, network context 200 includes a contextserver 205, an artificial intelligence service server 220, a clientdevice 230, and crowd-worker devices 225 a, 225 b and 225 c.

In the non-limiting example of FIG. 2, context server 205 is a physicalserver computer comprising, at a minimum, a processor, a memory, and anetwork interface for connecting to crowd workers 225 a-225 c and AIservice server 220. According to certain embodiments, context server 205receives crowd-sensed data from crowd-workers 225 a-225 c. Crowd-senseddata comprises data from which contextual signals may be extracted andincludes, without limitation, image data, video data, location data (forexample, Global Positioning System data). As will be described ingreater detail herein, in certain embodiments, context server 205utilizes crowd-sensed data from crowd workers 225 a-225 c to traincontextual labeling models (for example, models 210 in FIG. 2)implemented by context server 205. In other embodiments, context server205 also uses crowd-sensed data to build a corpus of contextual labeldata associated with locations in which crowd workers 225 a-225 c aredeployed. As a non-limiting example, context server 205 can build adatabase associating geographic locations (or GPS coordinates) withcontextual labels using image data from the geographic locationsprovided by 225 a-225 c.

Additionally, in certain embodiments, context server 205 cancrowd-source the implementation of models (for example, models 210 inFIG. 2) to generate predictions as to a contextual label to be assignedto a location. In certain embodiments, each crowd worker's prediction isreceived at context server 205, and the crowd workers' contextual labelpredictions are aggregated at context server 205.

According to various embodiments, context server 205 runs models 210 asan analytical adjunct to an artificial intelligence (AI) serviceprovided by a third party (for example, a service provided by AI serviceserver 220). In the non-limiting example of FIG. 2, models 210 are deepneural networks embodying a three layer schema, comprising a modellayer, an adaptation layer and an ensemble output layer. In certainembodiments, models 210 output probabilistic determinations that a givencontextual label is associated with received image data.

In some embodiments, annotation engine 215 acts as an ingestion pipelinefor aggregating crowd-sourced predictions of labels to be assigned toimages from a particular location.

In the non-limiting example of FIG. 2, AI service server 220 implementsor supports an artificial intelligence service. Examples of artificialintelligence services include, without limitation, intelligent assistantapplications, such as Bixby™, Alexa™, or Siri™, which can, in someinstances, apply artificial intelligence to parse and process userrequests, including, for example, verbal utterances recorded at asmartphone.

In certain embodiments, crowd workers 225 a-225 c are apparatus (forexample, apparatus 100 in FIG. 1) including processor, memory, send andreceive circuitry and one or more sensors capable of obtaininginformation (for example, image data, GPS coordinates, Wi-Fi networkidentities, etc.) which can be input into models 210. Examples ofdevices which can operate as crowd workers 225 a-225 c include, withoutlimitation, smartphones, tablets, smartwatches and laptop computers.

According to various embodiments, network context 200 includes a clientdevice 230. In the non-limiting example of FIG. 2, client device 230 isa device which passes a user command and, for example, location data(for example, a verbal request to the client device “Show me mytickets”) to AI service server 220. Applying contextual labelsassociated with the location data determined by models 210 (for example,a contextual label matching client device 230's location to an airport),AI service server 230 provides client device 230 with aprobabilistically determined execution option. In this case, executionoptions include, without limitation, opening an air travel applicationto show the user a plane ticket, or opening a digital wallet applicationcontaining a plane ticket.

In certain embodiments, client device 230 may also be a client worker.

Although FIG. 2 illustrates one example of a of a network context 200for implementing a deep-learning based, crowd-assisted system forcontextual labeling according to this disclosure, various changes may bemade to FIG. 2. For example, while network context 200 has beendescribed with respect to embodiments which perform contextual locationlabeling based on image data, the disclosure as a whole is not solimited. Location is merely one example of a useful contextual signalfor human-centric applications executing on apparatus according to thisdisclosure. Similarly, image data is merely one example of a source ofdata from which contextual labels may be determined according toembodiments of this disclosure. Other embodiments, using differentsources of data and contextual labels are possible and within theintended scope of this disclosure. For example, some embodiments maydetermine context from text in crowd sourced pictures and posts insocial networks to infer business names. Other embodiments may attemptto contextually categorize locations by training models on Wi-Fi or GSMdata. Still other embodiments may apply contextual labels (for example,“action movie fan”) based on models applied to content in a person'ssocial media feed.

Similarly, while FIG. 2 describes an embodiment employing aclient-server network architecture, the present disclosure is not solimited and embodiments according to this disclosure are generallyplatform-agnostic and may be implemented on a variety of scales andacross different network architectures. In some embodiments, contextuallabeling according to this disclosure may be performed on a singledevice, or in some embodiments, on a single chip. Similarly, in certainembodiments, elements shown in the illustrative example of FIG. 2 may beadded or subtracted. For example, in various embodiments, the separateAI service server 220 is omitted. In other embodiments, the AI servicemay be provided through context server 205.

FIG. 3 illustrates an example of a three-layer processing pipeline 300which includes a model adaptation schema and ensemble of adapted modelsaccording to this disclosure. As shown in FIG. 3, certain embodimentsaccording to this disclosure implement a three-pipeline for determiningcontextual classifiers for received data.

In the non-limiting example of FIG. 3, a three-layer pipeline 300 fordetermining contextual location data from visual data. A first layer 305comprises a set of pre-trained deep learning, or deep neural network(DNN) models. A second layer 310 comprises a model adaptation layer. Athird layer 315 comprises an ensemble output layer.

According to various embodiments, first layer 305 comprises one or morepretrained deep learning models 320. Further, in the non-limitingexample of FIG. 2, pretrained deep learning model 320 includes multiplelayers, which include a first layer 325 a, intermediate layers 325 b and325 c, and last layer 325 d. In some embodiments, for example,embodiments where no changes to the operation of the model are made,last layer 325 d may be considered part of first layer 305. In otherembodiments, last layer 325 d may be considered part of second layer310. The distinction between layers of pipeline 300 is to facilitateexplanation, and should not be interpreted as limitative of the scope ofthis disclosure.

As will be discussed in greater detail herein, last layer 325 d can be a“loss layer” or a layer implementing a SoftMax classifier. Layers 325 athrough 325 d can also include, without limitation, pooling layers,fully-connected layers and convolution layers.

In the non-limiting example of FIG. 3, deep learning model 320 ispre-trained to implement operations at each of layers 325 a through 325d to recognize learned deep features found in a data set, represent thedeep features with a feature vector and output probabilistic predictionsof specific classes of a model-specific domain being associated with therecognized deep features. Examples of DNN models which may beimplemented as pretrained deep learning model 320 include, withoutlimitation, the AlexNet and Inception-V3 models.

According to certain embodiments, deep learning model 320 is pre-trainedon a dataset specifically developed for an contextual labelingapplication (for example, determining contextual labels from imagedata). However, sufficiently training a deep neural network to avoidoverfitting frequently requires very large datasets and it may beimpractical to develop a dataset for training an entire deep neuralnetwork from scratch. Accordingly, in some other embodiments, it may bedesirable to instead pre-train deep learning model 320 on a very largestandard dataset (for example, ImageNet, which contains 1.2 millionimages with 1000 categories, or the Places dataset, which contains500,000 images with 205 categories) and use resulting model either as aninitialization or a fixed feature extractor to build a final model.According to still other embodiments, deep learning model is trained ona dataset generated by crowd worker devices (for example, crowd workers225 a-225 c in FIG. 2).

In some embodiments according to this disclosure, second layer 310comprises an adaptation layer.

According to some embodiments, the classifiers for each deep learningmodel in first layer 305 belong to a problem space (also referred to asa “source domain”) specific to the model. Depending on embodiments, thesource domains of the pre-trained models in first layer 305 may define adifferent classifier space to the space defined by the contextual labelsto be output by pipeline 300 (also referred to as a “target domain”).For example, first layer 305 may include a pre-trained deep learningmodel which outputs 1000 class labels, of which, only some arbitrarysubset are applicable to identifying a context of interest (for example,a location). In such cases, model adaptation layer 310 adapts theclassifier space of the deep learning model to the target domain.

According to certain embodiments, domain adaptation comprises labelspace adaptation, wherein the output of the final layer of a pretraineddeep learning model (for example, final layer 325 d shown in FIG. 3) isadapted without tuning learned parameters (weights) assigned to featurevectors or altering the internal structure of the model.

According to one non-limiting example, final layer 325 d of pretraineddeep learning model 320 implements a SoftMax classifier. The operationof final layer 325 d can, according to certain embodiments, berepresented by the following loss function:

$P( {y = { j \middle| X_{i}  = \frac{e^{w_{j}X_{i}^{T}}}{\sum\limits_{k = 1}^{n}e^{({w_{k}X_{i}^{T}})}}}} $

In the equation above, X_(i) is the feature vector extracted by the deepneural network for the input sample i (captured single image). w_(i) isthe weight learned by the neural network. y is the predicted class labelin j E N the set of all the class labels a pre-trained model is trainedon (the source domain).

In the non-limiting example of FIG. 3, the output of the SoftMaxclassifier can be adapted for a task of interest, including assigningcontextual labels in a target domain by following the Bayesian chainrule as shown below.

${P_{s}( {y =  j \middle| X_{i} } )} = \frac{1{( {y \in } ) \cdot {P( {y =  j \middle| X_{i} } )}}}{\sum_{}{1{( {y \in } ) \cdot {P( {y =  j \middle| X_{i} } )}}}}$

As shown above, 1(.) is the identity function and

is the label-set of the application the pre-trained model is adopted for(the target domain). The denominator is the normalization factor andthus P_(s)(y=j|X_(i)) indicates the probability of class (label) giventhe feature vector X_(i) for application specific labels j∈

. Thus, for a pre-trained model

with the label space

in the source domain, the above approach the model

for the target application with label space

⊂

.

According to some other embodiments, the differences between the spaceof the source domain and the target domain are such that modeladaptation following the Bayesian chain rule may not be possible.Examples of such embodiments include embodiments where the target domaincomprises a subset of the source domain (e.g., where

⊂

). One example of embodiments where

⊂

is the case where there are class labels in

which do not have any representation is the source domain (for example,where the label “computer store” is in the space

of the target domain, but there is no corresponding label for “computerstore” in the source domain). A further, example an embodiment where

⊂

includes instances where there are class labels in the space

of the target domain which match with multiple class labels in thesource domain. For example, in some embodiments, IL includes thecontextual location label “shoe-shop.” In cases where theAlexNet-ImageNet model defines the source domain

, there are multiple labels (for example, “shoe, “loafer shoe” and“sport shoe”).

In certain embodiments, adaptation layer 310 addresses situations where

⊂

by performing transfer learning. According to some embodiments, thefeature extracting, or convolutional layers (for example, layer 325 b)of pretrained deep learning model 320 are kept “frozen” by setting thelearning rate for those layers to zero. Additionally, a last fullyconnected layer of the model is initiated with random weights and thentrained on an additional data set labeled according to classifiers inthe space

. According to such embodiments, the previously trained featureextractors of the pretrained model are retained, while the final fullyconnected layers extend the model to cover the entirety of space

. According to some embodiments, such model extension allows fortraining a deep learning model using a limited amount of training data,while at the same, avoids overfitting. According to certain embodiments,besides utilizing the learning rate and structure of output layer 325 d,other network hyper-parameters are taken from the base model 320. In oneexemplary embodiment, a Rectified Linear Unit (ReLU) function is used asthe activation function in each convolution layer interleaved withpooling layers.

According to certain embodiments, third layer 315 of pipeline 300comprises an ensemble output layer. The accuracy with which pipeline 300predicts contextual labels for input data can, in certain embodiments,be enhanced by performing ensemble modeling of the outputs deep learningmodels 305 as adapted by adaptation layer 310. In some embodiments,performing ensemble modeling comprises determining a weighted average ofthe prediction probabilities of each of the pretrained models in firstlayer 305, as adapted or extended in second layer 310.

According to certain other embodiments, third layer 315 of pipeline 300further comprises aggregating the ensemble output from one or morepretrained deep learning models. For example, in certain embodiments,instead of running a single image from a location k, through pipeline300, a set of images from location k (represented as Γ_(k)) are sentthrough labeling pipeline 300. In such embodiments, the predictions foreach image i in Γ_(k) may be aggregated by applying the function:

${P_{\Gamma}( {y =  l \middle| \Gamma_{k} } )} = {\frac{1}{\Gamma_{k}}{\sum\limits_{X_{i}^{k} \in \Gamma_{k}}{P_{I}( {y =  j \middle| X_{i}^{k} } )}}}$

Wherein P_(I)(y=j|X_(i) ^(k)) is the prediction probability result byour ensemble of deep neural network models for classifying an image iobtained (for example, by a crowd-worker device) at location k,represented by feature vector X_(i) ^(k). As noted above, Γ_(k) is theset of all images obtained at location k, and P_(Γ)(y=l|Γ_(k)) is anaggregated prediction of the probability of label y across all imagesfor location k.

According to certain embodiments, once the aggregated probability ofeach label y across all images in Γ_(k) has been determined, a finallabel for each location k may be obtained by selecting the label withthe maximum aggregated probability. In one exemplary embodiment, thefinal label “label_(k)” may be selected by applying the function:

${label}_{k} = {\arg \; {\max\limits_{l}{P_{\Gamma}( {y =  l \middle| \Gamma_{k} } )}}}$

FIG. 4 illustrates operations of an exemplary method 400 for performingadaptation of pretrained deep learning models to obtain an ensembleoutput according to this disclosure.

As discussed elsewhere in this disclosure, method 400 may be practicedacross a variety of apparatus and networking contexts. In someembodiments, operations of method 400 may be practiced on a single chipin a single apparatus. According to other embodiments, the operations ofmethod 400 may be practiced across multiple machines, such as a networkof machines embodying a server-client paradigm.

In the non-limiting example of FIG. 4, method 400 is described withreference to providing an ensemble output of contextual labels based onimage data. However, the present disclosure is not so limited, andmethod 400 may be equally well applied to types of data containingcontextual signals other than image data.

According to certain embodiments, method 400 includes operation 405wherein an apparatus (as one non-limiting example, apparatus 100 inFIG. 1) implementing a processing pipeline (for example, pipeline 300 inFIG. 3) receives one or more pretrained deep learning models, each modelassociated with a source domain. As used herein, the term “sourcedomain” encompasses descriptions of a “source domain” from theperspective of an adaptation layer (for example, adaptation layer 310 inFIG. 3). For example, in some embodiments, at operation 405, anapparatus may receive AlexNet, which outputs one thousand classificationlabels. From the perspective of an adaptation layer, the one thousandclassification labels define a source domain for the model. Similarly,from the perspective of the adaptation layer, the full set of contextuallabels associated with possible outputs of method 400 comprise a targetdomain.

In some embodiments, method 400 includes operation 410, wherein theapparatus receives one or more items of image data to be contextuallylabeled. In some embodiments, the image data may be received from asingle source (for example, a camera on the apparatus implementing theprocessing pipeline). In other embodiments, the image data may bereceived from a plurality of sources (for example, crowd workers 225a-225 c in FIG. 2).

In the non-limiting example of FIG. 4, at operation 415, the receivedimage data is input into each of the one or more pretrained deeplearning models, thereby starting the process of passing the image datathrough the layers of the model to identify deep features within thedata and calculating the probability of labels in the source domainmatching the identified features.

According to some embodiments, method 400 includes operation 420,wherein the apparatus performs adaptation on each of the pretrained deeplearning models. According to some embodiments, such as where the labelspace IL of the target domain is a subset of the label space

for the target domain, (e.g., where

⊂

), adaptation may, as discussed elsewhere in this disclosure, beperformed by applying a Boolean filter to disregard labels in

which are not in

. In some embodiments, adaptation may be performed at operation 420 byapplying the function:

${P_{s}( {y =  j \middle| X_{i} } )} = \frac{1{( {y \in } ) \cdot {P( {y =  j \middle| X_{i} } )}}}{\sum_{}{1{( {y \in } ) \cdot {P( {y =  j \middle| X_{i} } )}}}}$

In at least one embodiment,

⊂

and at operation 425, the apparatus performs transfer learning on eachof the pretrained deep learning models which do not satisfy thecondition

⊂

. According to certain embodiments, performing transfer learningcomprises “freezing” the feature extracting, or convolutional layers, ofthe pretrained deep learning model 320 by setting the learning rate forthose layers to zero. Additionally, a last fully connected layer of themodel is initiated with random weights and then trained on an additionaldata set labeled according to classifiers in the space

. According to such embodiments, the previously trained featureextractors of the pretrained model are retained, while the final fullyconnected layers extend the model to cover the entirety of space

.

In certain embodiments, method 400 includes operation 430 wherein aresult in the target domain is obtained from each of the adaptedpretrained deep learning models.

According to some embodiments, at operation 435, the apparatus providesan ensemble output, wherein the ensemble output comprises a weightedaverage of prediction probabilities for labels in the target domainbased on the outputs from each of the adapted deep learning models. Inthe non-limiting example of FIG. 4, the weighting of outputs may beadjusted based on a confidence or uncertainty metric determined for eachof the adapted pretrained models.

FIG. 5 illustrates operations of a method 500 for performing adeep-learning based, crowd-assisted system for location labelingaccording to this disclosure.

While the non-limiting example of FIG. 5 describes one embodimentwherein all of the operations of method 500 are performed on a singleapparatus (for example, a smartphone or back-end server computer), thepresent disclosure is not so limited.

According to certain embodiments, method 500 includes operation 505wherein an apparatus receives from a user terminal, a control inputassociated with an intent. In certain embodiments, the apparatus is theuser terminal. In other embodiments, the apparatus may be a differentcomputing platform, such as, for example a back-end server connected viaa network (for example, the internet) to the user terminal. The controlinput may be received as a typed command, a gesture, or a verbalutterance associated with an execution option which can be performed bya processor of the user terminal. The intent associated with the controlinput, may in some embodiments, understood, or better understood withsome awareness of the context of the user terminal. As one non-limitingexample, a person may provide as a user input to her user terminal, thespoken query “What's the status of my order?” The user may also haveinstalled a number of applications on her user terminal which provideorder-related execution options (for example, a retail shoppingapplication, and an application associated with a coffee shop). In thisexample, the probability of selecting the execution option (for example,opening the retail application or opening the coffee shop application)associated with the control input is improved with an awareness of thecontext of the user terminal. If the user terminal can determine that itits current location is likely a coffee shop, the coffee shopapplication is likely the correct execution option.

In some embodiments, at operation 510, the apparatus obtains locationdata associated with the location of the user terminal. Location dataincludes, without limitation, image data, network connectivity data (forexample, an identification of active Wi-Fi hotspots at a given location)and GPS data. In this particular example, the term “location of the userterminal” encompasses the physical location of a user terminal (forexample, a smartphone) as a source of contextual signals as to theuser's intent. As discussed above, the knowledge that the user'ssmartphone is in a coffee shop provides a contextual signal that openingthe coffee shop application is likely the execution option best alignedwith the intent of the user's control input. However, the presentdisclosure is not so limited, and method 500 is equally operable wherethe “location of the user terminal” is a location in virtual, abstractor other non-terrestrial space. For example, a “location of the userterminal” may, for example, refer to the user's location in a graph of asocial network. In such cases, the user's location in the graph of thesocial network may provide contextual signals as to the execution optionbest matching the intent of the user's control input. For example, ifthe user input is a spoken request to “Play me a good movie,” knowledgeof the user's neighbors in a graph of a social network may providecontext in selecting a movie to play on the user terminal.

In various embodiments, at operation 515, the user terminal determines ascored set of execution options associated with the control input.Returning to the example of an apparatus receiving “what's the status ofmy order?” as a control input, at operation 515, the apparatusdetermines “open coffee shop application” and “open retail shoppingapplication” as members of the scored set of execution optionsassociated with the control input.

In the non-limiting example of FIG. 5, at operation 520, the apparatusobtains a contextual label associated with the location data, whereinthe label is determined based on the application of one or more adapteddeep learning models to image data associated with the location.According to certain embodiments, the contextual model is obtained byinputting the location data (in this case, image data associated withthe location of the user terminal) into a processing pipeline (forexample, pipeline 300 in FIG. 3) implemented on the apparatus. Further,the pipeline comprises one or more adapted, pretrained deep learningmodels. In this example, one or more of the pretrained deep learningmodels has been adapted by performing model extension using image dataobtained by crowd worker devices communicatively connected to thedevice. Further, in this non-limiting example, at least one of thepretrained deep learning models in the pipeline implemented at theapparatus is a lightweight, mobile-adapted version of a deep learningmodel.

Because deep learning models can be both computationally expensive andtake up a lot of storage space, implementing such models on smartphonesor other portable apparatus without rapidly draining batteries orconsuming storage resources required for important user content (forexample, photos, video and audio data) can present a technicalchallenge. In certain embodiments, the demands on the limited resourcesof mobile apparatus can be mitigated by “shrinking” the file size of themodel by applying a quantization method which takes advantage of theweights format of a trained model. Such “shrinking” can be attained by,for example, quantizing each 32 bit floating value in a model's weightmatrices to the closest (8 bit) integer number, resulting in anapproximately 75% reduction in file size.

According to certain embodiments, at operation 525, the apparatusrescores the execution options in the set of execution options based onthe contextual label obtained in operation 520. In one non-limitingexample, the contextual label associated with the location data was“coffee shop,” thereby raising the score for the execution option “opencoffee shop application.”

In the non-limiting example of FIG. 5, at operation 530, the apparatusidentifies, from the rescored execution options, the highest scoredexecution option and provides the highest scored execution option to aprocessor of the user terminal.

FIG. 6 illustrates, in tabular format, aspects of a pipeline of deeplearning models for generating an ensemble output according to thisdisclosure. The table in FIG. 6 is based on a set of deep learningmodels trained and adapted to provide an ensemble output predicting acontextual label associated with image data taken from retail locationsin a shopping mall. The performance results of the pipeline of deeplearning models shown in FIG. 6 are shown in FIG. 7 herein.

In the particular example of FIG. 6, the pipeline was developed based oneight separate deep learning models (shown as Models 1-8 in the columntitled “DNN Model”).

As shown in the column titled “Architecture,” Models 1-8 variedstructurally, with regard to the number of convolution and fullyconnected layers, and with regard to the number of elements in thesource domain, which ranged from 9 classes for “Model 8” to 1183 classesfor “Model 3.” In this particular example, the target domain comprised26 labels corresponding to, inter alia, different retail contexts (forexample, “shoe-store” and “supermarket”).

As shown in the column titled “Data Set,” Models 1-8 were generallytrained on large data sets having more classes than the 26 labels in thetarget domain. For models whose source domain, N, did not cover all ofthe 26 classes in the target domain, L, adaptation by transfer learningwas performed, as shown by the “→” icon in the “Data Set” column.

FIG. 7 illustrates, in five plots, performance data from a real-worldtest of the pipeline of deep learning models described with reference toFIG. 6. The test data shown in FIG. 7 is drawn from a test wherein thepipeline of pretrained deep learning models was implemented on asmartphone, and the ensemble outputs of the pipeline were aggregated onpredictions on crowd-sourced images. In each plot in FIG. 7, the name ofa store is specified (for example, “Safeway™”) and the pairings ofprobabilities and contextual labels for the top five predictionsproduced by the pipeline are shown in the plots. For example, for imagedata collected at a “Safeway™,” supermarket, the pipeline determinedthat there was a 68.52% probability of “supermarket” being the correctcontextual label for the image data.

FIG. 8 illustrates operations of a method 800 for generating alocation-device matrix according to certain embodiments of thisdisclosure. In the non-limiting example of FIG. 8, a location-devicematrix provides a probabilistic representation of a scene, or locationcorresponding to a context (for example, a “shoe store,” a “supermarket”or a “living room”) based on the application of convolutional neuralnetworks or deep learning networks to image data from the location toidentify devices within the location. According to certain embodiments,the location-device matrix serves as an input to a rescoring algorithmfor rescoring execution options based on contextual labeling.

According to certain embodiments, method 800 includes operation 805wherein an apparatus (for example, apparatus 100 in FIG. 1) parses imagedata associated with the location (for example, an image captured at thelocation) using a deep learning neural model. In some embodiments, thedeep learning model is a convolutional neural network (CNN). In otherembodiments, the deep learning model is a deep neural network (DNN).

In some embodiments, parsing image data at operation 805 comprisesidentifying features corresponding to devices (for example, legs, basesand other constituent parts of devices and objects in the scene). Fromthe identified devices, a vector representing objects in the scene (forexample, chairs, toasters and t.v. screens) can be compiled.

In the non-limiting example of FIG. 8, at operation 810, one or more ofthe deep learning models (for example a convolutional neural network) isadapted, so that all of its outputs belong to a target domain. Accordingto some embodiments, the convolutional neural network may be adaptedusing methods described in this disclosure (for example, the methodsdescribed with reference to operation 420 in FIG. 4).

In some embodiments, if the target domain is not completely covered bythe source domain associated with one or more convolutional neuralnetworks, transfer learning is performed at operation 815. In thenon-limiting example of FIG. 8, transfer learning may be performed usingmethods described in this disclosure (for example, the methods describedwith reference to operation 425 in FIG. 4).

According to various embodiments, at operation 820, the deep learningmodels (in this non-limiting example, the convolutional neural networks)provide an output in the source domain. Depending on embodiments, theoutput provided at operation 820 may be an output from a single item ofimage data. In other embodiments, the output provided at operation 820may be an aggregated output based on multiple pieces of image data fromthe location.

In some embodiments, the output in the target domain includes locationattention vector 825. In the non-limiting example of FIG. 8, locationattention vector 825 comprises the probabilities of various contextuallocation labels (for example, a “shoe store,” a “supermarket” or a“living room”) associated with the image data. As illustrated in FIG. 8,the various contextual location labels are shown with different fillpatterns. Thus, in this particular example, the upward diagonal fillpattern indicates the probability that the image corresponds to, thecontextual location label “supermarket,” while horizontal linescorrespond to the probability that the image corresponds to thecontextual location label “shoe store.”

According to certain embodiments at operation 830, a bipartite graphmapping locations to devices is updated based on the output of operation820. In some embodiments, the bipartite graph comprises a mapping of theedges between members of a set of contextual location labels anddevices. Further, according to some embodiments, each edge betweenlabels and devices is assigned a weighting based on the determinedcorrespondence between the contextual label and the device. Thus, atoperation 830, the edge weightings of the bipartite graph are updatedbased on the output of operation 820.

In the non-limiting example of FIG. 8, at operation 835, the apparatusbuilds location-device matrix 840 based on the bipartite graph updatedat operation 830. In the non-limiting example of FIG. 8, row of thelocation-device matrix comprises the weighting values associated betweencontextual location labels (for example “supermarket”) and devices (forexample, “toasters” or “tables”) parsed from image data associated withthe location.

According to some embodiments, the output of operation 835 compriseslocation-device matrix 840. As with location attention vector 825,upward diagonal cross-hatching is used to show entries oflocation-device matrix 840 associated with the contextual location label“supermarket.”

FIG. 9 illustrates aspects of building a location-device matrix 900based on a bipartite graph according to certain embodiments of thisdisclosure.

According to certain embodiments, an apparatus (for example apparatus100 in FIG. 1) implementing methods according to this disclosuremaintains and updates a bipartite graph 905 of contextual labels anddeep features parsed from a contextual signal-rich source of data. Inthe non-limiting example of FIG. 9, contextual labels comprisecontextual location labels 910 a and 910 b. Further, in this particularexample, the parsed features comprise devices 915 a, 915 b and 915 crecognized from image data through the application of one or moreconvolutional neural networks.

As shown in FIG. 9, in bipartite graph 905, edges between each elementof the set of contextual location labels and each element of the set ofdevices are determined. Further, as shown with respect to edge 920, eachedge is associated with a weight quantifying the correspondence betweena device and a contextual label. In this non-limiting example, edge 920connecting contextual label l₂ to device d₃ is shown as having aweighting value of −0.9, indicating a negative correspondence between adevice and a contextual location label. In this example, a largenegative weighting value indicates that the presence of a particulardevice (for example, “basketball hoop”) corresponds to a particularcontextual location label (for example, “supermarket”).

As shown in the non-limiting example of FIG. 9, each of the weightingvalues of edges between elements of a set of contextual labels andelements of a set of features become values of the location-devicematrix 900. As shown by matrices 900 and 925, rows of thelocation-device matrix the weightings between devices and a particularcontextual label. For example, in location-device matrix 925, the rowshown with a diagonal fill corresponds to the contextual location label“supermarket.” According to some embodiments, the output of operation835 comprises location-device matrix 840.

FIG. 9 illustrates aspects of building a location-device matrix 900based on a bipartite graph according to certain embodiments of thisdisclosure.

According to certain embodiments, an apparatus (for example apparatus100 in FIG. 1) implementing methods according to this disclosuremaintains and updates a bipartite graph 905 of contextual labels anddeep features parsed from a contextual signal-rich source of data. Inthe non-limiting example of FIG. 9, contextual labels comprisecontextual location labels 910 a and 910 b. Further, in this particularexample, the parsed features comprise devices 915 a, 915 b and 915 crecognized from image data through the application of one or moreconvolutional neural networks.

As shown in FIG. 9, in bipartite graph 905, edges between each elementof the set of contextual location labels and each element of the set ofdevices are determined. Further, as shown with respect to edge 920, eachedge is associated with a weight quantifying the correspondence betweena device and a contextual label. In this non-limiting example, edge 920connecting contextual label l₂ to device d₃ is shown as having aweighting value of −0.9, indicating a negative correspondence between adevice and a contextual location label. In this example, a largenegative weighting value indicates that the presence of a particulardevice (for example, “basketball hoop”) corresponds to a particularcontextual location label (for example, “supermarket”).

As shown in the non-limiting example of FIG. 9, each of the weightingvalues of edges between elements of a set of contextual labels andelements of a set of features become values of the location-devicematrix 900. As shown by matrices 900 and 925, rows of thelocation-device matrix the weightings between devices and a particularcontextual label.

FIG. 10 illustrates aspects of an algorithm 1000 for rescoring executionoptions according to certain embodiments of this disclosure.

In the non-limiting example of FIG. 10, the inputs of algorithm 1000 forrescoring execution options based on contextual location labels compriseinitial intelligent assistant determination 1005, attention vector 1010,location-intention matrix 1015, location-device matrix 1020, and devicecapability matrix 1025.

According to certain embodiments, initial intelligent assistantdetermination 1005 determination 1005 comprises a vector ofprobabilities mapped to execution options of devices at a user'slocation, as initially determined by an intelligent assistant (forexample AI service 220) based on the intelligent assistant's analysis ofthe user's intention from a control input, without regard to anycontextual labeling. In the non-limiting example of FIG. 10, the vectorcomprising initial intelligent assistant determination 1005 may berepresented in matrix form as u_(u*1).

As a simple example, a user provides the spoken input “What's the statusof my order?” The possible execution options associated with the userinput include “open coffee shop application” and “open retail shoppingapplication.” According to some embodiments, initial intelligentassistant determination 1005 may comprise a vector having values of 0.3for the execution options “open coffee shop application” and “openretail shopping application.” In this example values of the vector areprobabilistic scores as to whether an execution option correctly alignswith the user's intention. In this particular example, the apparatus isable to determine that both of these options are likely, but does nothave any contextual information to prefer the coffee shop applicationover the retail shopping application.

In some embodiments, attention vector 1010 comprises, an ensemble outputof a pipeline (for example, pipeline 300 in FIG. 3) into which imagedata from a location i, has been inputted. According to someembodiments, attention vector 1010 comprises an aggregation of ensembleoutputs for multiple pieces of image data from location i. Theconstituent members of attention vector 1010 are the probabilities, thatimage data from location i is associated with a particular contextuallabel j. In the non-limiting example of FIG. 10, a_(ij) is representedas P(Location_(i)\Input Image_(x)).

According to various embodiments, location-intention matrix 1015comprises a matrix of probabilities of execution options (or userintentions) across contextual location labels. In the non-limitingexample of FIG. 10, the constituent entries Q_(ij) of location-intentionmatrix 1015 may be represented as P(Intention_(i)\Location_(j)).

In the non-limiting example of FIG. 10, location-device matrix 1020comprises a location-device matrix (for example, location-devicematrices 900 and 925 in FIG. 9). In the non-limiting example of FIG. 10,the constituent elements M_(ij) of location-device matrix 1020 may berepresented as P(Device_(i)\Location_(j)).

According to certain embodiments, device capability matrix 1025comprises a mapping of probabilities that a device obtained by parsingimage data for a location i is associated with a particular executionoption or user intent. In the non-limiting example of FIG. 10, theconstituent members R_(ij) of device capability matrix 1025 may berepresented as P(Intention_(i)\Device_(j)).

According to certain embodiments, the scores comprising vector u_(u*1)comprising initial intelligent assistant determination 1005 can berecalculated, or “rescored” by applying the function R(u) as shownbelow:

R(u)=[Q.conc(a)]^(T) ·M ^(T) ·Y ^(T) ·u

Wherein M^(T) is the transpose of location-device matrix 1020),[Q.conc(a)]^(T) is the transpose of the dot product of theuser-intention location matrix Q, and the concatenation of locationattention vector a 1010, and Y^(T) is the transpose of device-capabilitymatrix 1025.

Application of R(u) results in the vector 1030, which is a vector in thesame space as initial intelligent assistant determination 1005, but inwhich the constituent probabilities of user intentions are rescoredbased on contextual information provided by, without limitation,location-device matrix 1020.

None of the description in this application should be read as implyingthat any particular element, step, or function is an essential elementthat must be included in the claim scope. The scope of patented subjectmatter is defined only by the claims. Moreover, none of the claims isintended to invoke 35 U.S.C. § 112(f) unless the exact words “means for”are followed by a participle.

What is claimed is:
 1. An apparatus for contextual labeling of imagedata, the apparatus comprising: a processor; and a memory containinginstructions, which when executed by the processor, cause the apparatusto: receive one or more pretrained deep learning models, each pretraineddeep learning model associated with a source domain, receive image datato be labeled, input the received image data to each of the one or morepretrained deep learning models, perform an adaptation on one or more ofthe pretrained deep learning models, provide, from each of thepretrained deep learning models, an output in a target domain, providean ensemble output, the ensemble output comprising labels for the imagedata determined based on the outputs from each of the pretrained deeplearning models, and when the target domain is not completely covered bythe source domain associated with a pretrained deep learning model ofthe one or more pretrained deep learning models, perform transferlearning on the pretrained deep learning model.
 2. The apparatus ofclaim 1, wherein performing transfer learning further comprisesperforming a model extension by training a final layer of the pretraineddeep learning model for which the target domain is not completelycovered by the source domain.
 3. The apparatus of claim 1, whereinperforming the adaptation on one or more pretrained deep learning modelscomprises applying the function${P_{s}( {y =  j \middle| X_{i} } )} = {\frac{1{( {y \in } ) \cdot {P( {y =  j \middle| X_{i} } )}}}{\sum_{}{1{( {y \in } ) \cdot {P( {y =  j \middle| X_{i} } )}}}}.}$4. The apparatus of claim 1, the memory further including instructions,which when executed by the processor, cause the apparatus to: aggregatethe ensemble output of each pretrained deep learning model of the one ormore pretrained deep learning models by applying the function${P_{\Gamma}( {y =  l \middle| \Gamma_{k} } )} = {\frac{1}{\Gamma_{k}}{\sum\limits_{X_{i}^{k} \in \Gamma_{k}}{{P_{I}( {y =  j \middle| X_{i}^{k} } )}.}}}$5. The apparatus of claim 4, the memory further including instructions,which, when executed by the processor, cause the apparatus to: labelimage data for a location k by applying the function${label}_{k} = {\arg \; {\max\limits_{l}{{P_{\Gamma}( {y =  l \middle| \Gamma_{k} } )}.}}}$6. A method for contextual labeling of image data, the methodcomprising: receiving, one or more pretrained deep learning models, eachpretrained deep learning model associated with a source domain;receiving image data to be labeled; inputting the received image data toeach of the one or more pretrained deep learning models; performing anadaptation on one or more of the pretrained deep learning models;providing, from each of the pretrained deep learning models, an outputin a target domain for the pretrained deep learning model; providing anensemble output, the ensemble output comprising labels for the imagedata determined based on the outputs from each of the pretrained deeplearning models; and when the target domain is not completely covered bythe source domain associated with a pretrained deep learning model ofthe one or more pretrained deep learning models, perform transferlearning on the pretrained deep learning model.
 7. The method of claim6, wherein performing transfer learning further comprises performing amodel extension by training a final layer of the deep learning model forwhich the target domain is not completely covered by the source domain.8. The method of claim 6, wherein performing the adaptation on each ofthe one or more pretrained deep learning models comprises applying thefunction${P_{s}( {y =  j \middle| X_{i} } )} = {\frac{1{( {y \in } ) \cdot {P( {y =  j \middle| X_{i} } )}}}{\sum_{}{1{( {y \in } ) \cdot {P( {y =  j \middle| X_{i} } )}}}}.}$9. The method of claim 6, further comprising aggregating the ensembleoutput of each pretrained deep learning model of the one or morepretrained deep learning models by applying the function${P_{\Gamma}( {y =  l \middle| \Gamma_{k} } )} = {\frac{1}{\Gamma_{k}}{\sum\limits_{X_{i}^{k} \in \Gamma_{k}}{{P_{I}( {y =  j \middle| X_{i}^{k} } )}.}}}$10. The method of claim 9, further comprising labeling image data for alocation k by applying the function${label}_{k} = {\arg \; {\max\limits_{l}{{P_{\Gamma}( {y =  l \middle| \Gamma_{k} } )}.}}}$11. The method of claim 6, further comprising, updating a bipartitegraph of the elements of the target domain to devices.
 12. Anon-transitory computer-readable medium including program code, whichwhen executed by a processor, causes an apparatus to: receive one ormore pretrained deep learning models, each pretrained deep learningmodel associated with a source domain, receive image data to be labeled,input the received image data to each of the one or more pretrained deeplearning models, perform an adaptation on one or more of the pretraineddeep learning models, provide, from each of the pretrained deep learningmodels, an output in a target domain for the pretrained deep learningmodel, provide an ensemble output, the ensemble output comprising labelsfor the image data determined based on the outputs from the pretraineddeep learning models, and when the target domain is not completelycovered by the source domain associated with a pretrained deep learningmodel of the one or more pretrained deep learning models, performtransfer learning on the pretrained deep learning model.
 12. Thenon-transitory computer-readable medium of claim 12, wherein performingtransfer learning further comprises performing a model extension bytraining a final layer of the deep learning model for which the targetdomain is not completely covered by the source domain.
 13. Thenon-transitory computer-readable medium of claim 12, wherein performingthe adaptation on each of the one or more pretrained deep learningmodels comprises applying the function${P_{s}( {y =  j \middle| X_{i} } )} = {\frac{1{( {y \in } ) \cdot {P( {y =  j \middle| X_{i} } )}}}{\sum_{}{1{( {y \in } ) \cdot {P( {y =  j \middle| X_{i} } )}}}}.}$14. The non-transitory computer-readable medium of claim 12, furthercomprising program code, which when executed by the processor, cause theapparatus to aggregate the ensemble output of each pretrained deeplearning model of the one or more pretrained deep learning models byapplying the function${P_{\Gamma}( {y =  l \middle| \Gamma_{k} } )} = {\frac{1}{\Gamma_{k}}{\sum\limits_{X_{i}^{k} \in \Gamma_{k}}{{P_{I}( {y =  j \middle| X_{i}^{k} } )}.}}}$15. The non-transitory computer-readable medium of claim 15, furthercomprising program code, which when executed by the processor, causesthe apparatus to label image data for a location k by applying thefunction${label}_{k} = {\arg \; {\max\limits_{l}{{P_{\Gamma}( {y =  l \middle| \Gamma_{k} } )}.}}}$17. The non-transitory computer-readable medium of claim 12, furthercomprising, program code, which when executed by the processor, causesthe apparatus to update a bipartite graph of the elements of the targetdomain to devices.
 18. An apparatus for contextual execution, theapparatus comprising: a processor; and a memory containing instructions,which when executed by the processor, cause the apparatus to: receive,from a user terminal, a control input associated with an intent, obtainlocation data associated with a location of the user terminal, determinea scored set of execution options associated with the control input,obtain a contextual label associated with the location data, thecontextual label determined based on application of one or more adaptedpretrained deep learning models to the location data, rescore the set ofexecution options associated with the control input based on thecontextual label, and provide the highest scored execution option to aprocessor of the user terminal.
 19. A method for contextual execution,the method comprising: receiving, from a user terminal, a control inputassociated with an intent, obtaining location data associated with alocation of the user terminal, determining a scored set of executionoptions associated with the control input, obtaining a contextual labelassociated with the location data, the contextual label determined basedon application of one or more adapted pretrained deep learning models tothe location data, rescoring the set of execution options associatedwith the control input based on the contextual label, and providing thehighest scored execution option to a processor of the user terminal. 20.A non-transitory computer-readable medium including program code, whichwhen executed by a processor, causes an apparatus to: receive, from auser terminal, a control input associated with an intent, obtainlocation data associated with a location of the user terminal, determinea scored set of execution options associated with the control input,obtain a contextual label associated with the location data, thecontextual label determined based on application of one or more adaptedpretrained deep learning models to the location data, rescore the set ofexecution options associated with the control input based on thecontextual label, and provide the highest scored execution option to aprocessor of the user terminal.